(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 




(19) World Intellectual Property Organization 

International Bureau 

(43) International Publication Date (10) International Publication Number 

25 April 2002 (25.04.2002) pCT WO 02/33892 A2 



(51) International Patent Classification'^: 



H04L 12/00 



(21) International Application Number: PCTAJSO 1/3 1420 

(22) International Filing Date: 4 October 2001 (04.10.2001) 



(25) Filing Language: 

(26) Publication Language: 



English 



English 



(30) 



(63) 



Priority Data: 

60/241,450 
60/275,206 
09/903,423 



17 October 2000 (17.10.2000) US 
12 March 2001 (12.03.2001) US 
10 July 2001 (10.07.2001) US 



Related by continuation (CON) or continuation-in-part 
(CIP) to earlier applications: 

US 60/241,450 (CON) 

Filed on 17 October 2000 (17. 10.2000) 

US 60/275,206 (CON) 

Filed on 12 March 2001 (12.03.2001) 

US 09/903,423 (CON) 

Filed on 10 July 2001 (10.07.2001) 



(71) Applicant (for all designated States except US): ROUTE- 
SCIENCE TECHNOLOGIES, INC. [US/US]; 167 2nd 
Avenue, San Mateo, CA 94401 (US). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): LLOYD, Michael, 
A. [US/US]; 160 Arundel Road, San Carlos, CA 94070 
(US). FINN, Sean, P. [US/US]; 1533 Escondido Way, 
Belmont, CA 94002 (US). BALDONADO, Omar, C. 
[US/US]; 700 Alester Avenue, Palo Alto, CA 94303 
(US). KARAM, Mansour, X [LB/US]; 707 Continental 



Circle, #421, Mountain View, CA 94040 (US). SIDDIQI, 
Faisal [IN/US]; 85 West 5th Avenue, #502, San Mateo, 
CA 94402 (US). MCGUIRE, James, G. [US/US]; 2312 
Gough Street, San Francisco, CA 94019 (US). MADAN, 
Herbert, S. [US/US]; 347 Blackfield Drive, Tiburon, CA 
94920 (US). 

(74) Agent: SUZUE, Kenta; Wilson Sonsini Goodrich & 
Rosati, 650 Page Mill Road, Palo Alto, CA 94304-1050 
(US). 

(81) Designated States (national): AE, AG, AL, AM, AT, AU, 

AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU, 
CZ, DE, DK, DM, DZ, EC, EE, ES, K, GB, GD, GE, GH, 
GM, HR, HU, ID, IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, 
LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, 
MX, MZ, NO, NZ, PH, PL, PT, RO, RU, SD, SE, SG, SI, 
SK, SL, TJ, TM, TR, TT, TZ, UA, UG, US, UZ, VN, YU, 
ZA, ZW. 

(84) Designated States (regional): ARIPO patent (GH, GM, 
KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZW), Eurasian 
patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), European 
patent (AT, BE, CH, CY, DE, DK, ES, FI, FR, GB, GR, IE, 
IT, LU, MC, NL, PT, SE, TR), OAPI patent (BE, BJ, CF, 
CG, CI, CM, GA, GN, GQ, GW, ML, MR, NE, SN, TD, 
TG). 

Published: 

— without international search report and to be republished 
upon receipt of that report 

For two-letter codes and other abbreviations, refer to the "Guid- 
ance Notes on Codes and Abbreviations" appearing at the begin- 
ning of each regular issue of the PCT Gazette. 



< 

^ (54) Title: SYSTEMS AND METHODS FOR ROBUST, REAL-TIME MEASUREMENT OF NETWORK PERFORMANCE 

o\ 

2? (57) Abstract: Methods and apparatuses for obtaining delay, jitter, and loss statistics of a path between server and an end user 
coupled via an internetwork are described. The serve may comprise a web server in communication with the end user via the Internet. 
^ Statistics are obtained by analyzing the details of a TCP connection underlying an HTML transaction. Robust measurements of jitter, 
delay, and loss are ensured by maximizing traffic between the web server and the surfer in order to generate a robust sample of TCP 
connections. Content may be updated with one or more html link(s). This existing content may reside on a highly trafficked portal, 
such as a web portal, and may be encoded in a markup language, such as Hyper Text Markup Language (HTML). The Uniform 
Resource Locators (URLs) corresponding to the one or more links resolve to the server from which the statistics are to be measured. 
The actual content supplied by the server may be minimized, in order to preserve bandwidth. 
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SYSTEMS AND METHODS FOR ROBUST, REAL-TIME 

MEASUREMENT 
OF NETWORK PERFORMANCE 

5 Description of Related Art 

The performance characteristics of routes In internetworks, such 
as the Internet, have been assessed In prior efforts. Statistical metrics 
of Internet performance include the characteristics of jitter, loss, and 
delay. Jitter may be characterized as the amount of variance in the time 
10 taken by packets traversing a path In a network. Delay indicates the 
amount of time taken for packets to traverse the path. And loss 
Indicates the lossiness of the internetwork path. 

Empirical observations have demonstrated that various 
combinations of these performance metrics are especially relevant to the 
15 performance of certain types of applications on the Internet. For 

instance, in some voice streaming applications such as Voice over IP 
(VoIP), appreciable levels of jitter may have a highly deleterious effect 
on performance, while some packet loss may be tolerable. In other 
applications, jitter and delay may be tolerable, while significant packet 
20 loss may be fatal. 

Given the significance of such metrics to Internet performance, 
' * there Is a need to measure such statistics In real-time for arbitrary end- 
points in an internetwork. The prior art also evinces a need to ensure 
that such statistics are robust, and based on substantial packet traffic. 

25 

Summary of the Invention 

Some embodiments of the Invention Include methods and 
apparatuses for obtaining delay, jitter, and loss statistics of a path 
between server and an end user coupled via an internetwork; in some 
30 embodiments, the server may comprise a web server in communication 
with the end user via the Internet. In some embodiments of the 
invention, these statistics are obtained by analyzing the details of a TCP 
connection underlying an HTML transaction. Some such embodiments 

1 
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ensure robust measurements of jitter, delay, and loss by maximizing 
traffic between the web server and the surfer In order to generate a 
robust sample of TCP connections. 

In some such embodiments, content is updated with one or more 
5 html link(s). This existing content may reside on a highly trafficked 
portal, such as a web portal, and may be encoded in a markup 
language, such as Hyper Text Markup Language (HTML). The Uniform 
Resource Locators (URLs) corresponding to the one or more links 
resolve to the server from which the statistics are to be measured, i.e., 

1 0 the server which connects to the end user over the desired path. In 
some embodiments, this resolution may be based on an explicit 
relationship between a URL and a given measurement path. In 
alternative embodiments, the one or more URLs may resolve to an 
address which varies on each invocation, such that only the address, 

15 rather than the URL, connotes a relationship with the specific 

measurement path. A request for the connection comes Into the server, 
and based on the target address, the outbound response is 
subsequently forced to a specific measurement path. In some 
embodiments of the invention, the actual content supplied by the server 

20 is minimized, in order to preserve bandwidth. In some embodiments, the 
content may be visually imperceptible, comprising one or more pixels, 
which may be transparent. In other embodiments, the content may 
comprise a visual artifact 

Some embodiments of the invention include a measurement 

25 subsystem which records observed call response times, which are used 
to record round trip times for packets traversing the path between the 
server and the end user. In some embodiments, these packets employ 
the TCP/IP protocol for their transport. In alternative embodiments, 
these measurements may be gathered at the end-user side, as opposed 

30 to the server side. 

Some embodiments of the invention measure round trip times for 
different patterns of TCP messages sent within a TCP connection. In 
some embodiments, these measurements of round trip times are 



converted into measurements of jitter, loss, or delay along the desired 
path. In some embodiments of the invention, jitter, loss, and delay 
statistics may be inferred by groups, or classes, of end user addresses. 
These and other embodiments are further described herein. 

Brief Description of Figures 

Figure 1 illustrates an architecture used to redirect internetwork 
traffic to a measurement server according to some embodiments of the 
invention. 

Figure 2 illustrates techniques used to measure Round Trip 
Times for various types of TCP sessions according to some 
embodiments of the invention. 

Detailed Description 

Distributing Hits to the Desired Server 

Some embodiments of the invention include systems and 
methods to maximize traffic through a desired path, in order to generate 
a robust number of measurements of round trip times through the path. 
These embodiments are illustrated schematically in Figure 1 . The 
method generates traffic towards an end user 102, or surfer. An 
internetwork 100 includes a measured server 104, which is the server 
from which traffic is to be measured, and a highly trafficked portal 106. 
The highly trafficked portal 106 may include content from a popular 
commercial web site. The measured server and the end user can 
communicate via the internetwork through one or more paths 108. 
Such embodiments attempt to divert traffic from the portal 106 to the 
measured server 104, in order to ensure robust measurements of 
network performance along the one or more paths 108. 

In some such embodiments, a content object is included in the 
portal 106, so that when an end user 102 connects to the portal 106, her 
request is redirected to the measured server 104 in order to receive the 
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portion of content. This content object may be referred to as a webby. 
In some embodiments of the invention, the webby is designed to occupy 
a minimal amount of bandwidth. In some embodiments, the webby is 
designed to be imperceptible. In a non-limiting implementation of the 
5 webby, the content object may comprise a transparent GIF or JPEG, 
which includes one or more pixels. Other implementations of the 
content object will be apparent to those skilled in the art. 

In web based embodiments, when a surfer's browser 102 
requests the content object, the browser 102 performs a DNS lookup, 

10 and retrieves an IP address for the web object; this IP address resolves 
to the measured server 104. In some embodiments of the invention, by 
supplying varying answers for the IP address, hits may be distributed 
across many measured servers 104. In response to the request, the 
measured server 104 delivers the content object to the surfer's browser 

15 102. 

Measuring Round Trip Times 

Some embodiments of the invention measure Round Trip Times 
(RTTs) between the measured server 104 and end users 102 in order to 

20 generate metrics of path performance; these metrics may, by way of 
non-limiting example, include jitter, delay, and loss statistics. In some 
embodiments of the invention, different algorithms for measuring RTTs 
are employed, contingent upon the type of session that is witnessed. As 
such, several types of TCP sessions are described herein, followed by a 

25 discussion of the RTT measurement techniques that may be employed 
for the various sessions. Note that the discussion that follows employs 
acronyms described in Table 1 below: 
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Table 1 Acronyms used in the description of TCP patterns 



Si 


SYN received by the webby (i.e., incoming SYN) 


So 


SYN/ACK sent by the webby 


Pi 


PUSH packet received by the webby 


Po 


PUSH packet sent by the webby 


Fi 


FIN message received by the webby 


Fo 


FIN message sent by the webby 


.i 


ACK message received by the webby 


.0 


ACK message sent by the webby 



Figure 2 illustrates three types of sessions 200 202 204 that may 
5 be witnessed between the measured server 1 04 and the end user, or 
surfer 102. These patterns are hereafter referred to as Basic Pattern 1 
(B1) 200, Basic Pattern 2 (B2) 202, and Basic Pattern 3 (B3) 204. The 
differences between patterns B1 on one hand, B2 and B3 on the other, 
inheres in the manner in which TCP behaves on the side of the webby, 
10 i.e., the measured server 104. In the case of B1 200, the actions 

performed by the webby 104 upon the receipt of a PUSH packet (i.e., Pi) 
are as follows: 

• The webby 104 sends an ACK packet acknowledging the PUSH. 

• The webby 104 sends the requested data in a PUSH packet. 
15 • The webby 104 subsequently terminates the connection by 

sending a FIN message. 
For cases B2 202 and B3 204, the actions performed by the webby 104 
upon the receipt of a PUSH packet (i.e., Pi) are as follows: 

• The webby 104 sends an ACK packet acknowledging the PUSH 
20 • The webby 104 sends the requested data in a PUSH packet 

• The webby 104 subsequently waits for an acknowledgment from 
the surfer 102 containing notification of receipt of the data before 
the webby 104 proceeds with sending a FIN, 
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In some embodiments of the invention, Round Trip Times RTT^, 
RTT2 and RTT3 are computed by use of the same algorithm in all cases 
200 202 204. In some such embodiments, RTT^ may be determined 
simply by waiting for an ACK corresponding to the first SYN/ACK. In 
5 some embodiments, RTT2 may be measured by starting a timer at the 
instant the first PUSH is sent by the webby 104 (as for RTT^, the timer is 
started at the first PUSH to take into account the effect of timeouts), and 
stopping the timer upon the receipt of the first packet acknowledging the 
PUSH that was sent. (This packet acknowledges a sequence number at 

10 least equal to that of the PUSH message). A similar technique may be 
applied to RTT3, this time to the FIN packet sent by the webby 104. As 
discussed in U.S. Provisional Applications 60/241,450, filed October 17, 
2000 and 60/275,206, filed March 12, 2001, which are hereby 
incorporated by reference in their entirety, these techniques for 

1 5 measuring Round Trip Times have been empirically shown to be robust 
in all manner of complex TCP transactions. 

Computation of Jitter. Loss, and Delay from Round Trip Times 

In some embodiments of the invention, a measurements listener 
20 receives values of RTTi, RTT2, and RTTz that correspond to a given IP 
address. In some embodiments, the measurements listener may 
comprise one or more processes distributed on one or more servers 
coupled to the internetwork. These measurements are sent to a module 
that peri'orms one or more of the following steps: 
25 • Compute the values of round-trip time d, jitter v, and packet 

loss p for this measurement Instance. 
• Map the IP address to a corresponding group of IP 

addresses (this group may comprise an Equivalence Class, 
which is further described in which are hereby incorporated by 
30 reference in their entirety) 
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• Update the values ofd, v , p , using old values of d , v, p 

and the values of of, v, and p, wherein a , v, p comprise 
weighted averages of delay, jitter, and loss, respectively. 

Non-limiting implementations for calculating d, v, and p from the 
5 Round Trip Times are described herein. First, note that RTT^ and RTT3 
do not overlap in some embodiments. Hence, network events that are 
captured by the first round trip time RTTi are typically not captured by 
RTTz. Empirical observations also demonstrate that RTT^ and RTTz are 
often very different. As such, some embodiments of the invention 
10 employ a difference between RTT^ and RTT3 to capture network 

oscillations in performance, i.e. jitter. In one such embodiment the jitter, 
V is set to the absolute value of the difference, i.e., 

V = \RTTz-RTT^\ 

15 Empirical observations also demonstrate that RTT2 and RTTz 

may be highly correlated. As such, in some embodiments of the 
invention a difference between RTTz and RTTz may be used to infer 
packet loss. In case RTTz is not measured, a large difference between 
RTTi and R7T2 may be used to infer packet loss in extreme cases, for 
20 example when RTT^ is close to 0, and RTT2 has a value on or about 3 
seconds. Otherwise, a difference between RTT2 and RTTz that is close 
to 3 or 6 seconds may be used in some embodiments of the invention, to 
declare packet loss. Thus, to determine loss, some embodiments of the 
invention employ one or more of the following steps: 
25 • If either RTT^ or RTT2 is small (for example, less than 

500 ms), compute the difference between R7Ti and 
RTT2: if this difference is on or about 3 seconds or 6 
seconds, set p to 1 . 
• If either RTT^ or RTT2 is large (for example, more than 
30 500 ms), compute the difference between RTT2 and 

RTTz: if this difference Is on or about 3 seconds or 6 
seconds, set p to 1. 
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• Otherwise set p to 0. 

In some embodiments of the invention, of Is set to an average of 
the true RTTs measured for a transaction. In case p is set to 0, this is 
5 simply the average of all three RTTs. In case p is set to 1, the packet 
involved in the loss should be removed from the computation of the 
average d. (Alternatively, a 3 second timeout can be subtracted from the 
measured RTT for that packet.) 

As will be apparent to those skilled in the art, the implementations 
10 described are non-limiting techniques for computing d, v, and p from 
Round Trip Times; other implementations will be apparent to those 
skilled in the art. 

Computing Weighted Averages of Jitter. Delay, and Loss 

16 

Some embodiments of the invention include techniques for 

maintaining weighted averages of Delay, Jitter, and Loss. ^, v, and P 
respectively. In some such embodiments, current values of of, v, and p 

values as well as previous values of d ^ v , and P for a relevant group of 

20 IP addresses are used to compute the new values for , v , and P . 
In a non-limiting example, weighted moving averages are used to 

compute ^ , V , and P 

Pne^,^JPold+{^-r)p 

25 In some embodiments, a, p, and y are fixed constants. In some 

such embodiments, the combination of values used for a, p, and y are 
determined by the type of application the TCP session is supporting. 
These applications may include, but are not limited to, any one or more 
of HTTP 1 .0, HTTP 1 .1 , Voice over IP, or Video streaming over IP. 
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Examples of values of a, p, and y that may be used for these 
applications are presented below in an XML format. Note that these 
examples also include sample values for parameters denoted theta, phi, 
omega, and psi; these parameters may be used to convert the tuples (a, 
5 p, and y) into a scalar performance score; these parameters are further 
described in U.S. Provisional Applications 60/241,450, filed October 17, 
2000 and 60/275,206, filed March 12, 2001, which are hereby 
incorporated by reference in their entirety. The values presented herein 
are for illustration only; other value combinations will be apparent to 
10 those skilled in the art: 

HTTP 1-0 

<module> <engine slot="1"> <application model="http1.0" [alpha="0.9" 
beta="0.9" gamma="0.9" theta="1,18" phi="0.13" omega="0.15" 
15 psi="0.25"] /> </engine> </module> 

HTTP 1.1 

<module> <engine slot="1"> <application model="http1.1" [alpha="0.9" 
beta="0.9" gamma="0.9" theta="1,3" phi="0.31" omega="0.41" psi="1.0"] 
20 /> </engine> </module> 

Voice over IP 

<module> <engine slot="1"> <appllcation model="voice" [alpha- '0.9" 
beta="0.9" gamma="0.9" theta ="1.5" phi="6.0" omega="23.0" psi="0.0"] 
25 /> </engine> </module> 

Video Streaming 

<module> <engine slot="1"> <application model="video" [alpha="0.9" 
beta="0.9" gamma="0.9" theta="1 ,0" phi="4.0" omega="69.0" psi="0.0"] 
30 /> </engine> </module> 
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In some embodiments of the invention, time-decaying values of a, 
p, and Y may be employed. In some such embodiments, these values of 
a, p, and y may decay exponentially, i.e., 
a = exp(-ka T) 
5 p = exp(-kp T) 

y = exp(-kyT) 

Other value combinations for a, p, and y shall be apparent to 
those skilled in the art. 

10 Conclusion 

The various techniques presented above for measuring Round 
Trip Times and determining jitter, loss, and delay values are presented 
for illustrative purposes only. Many equivalent techniques shall be 
apparent to those skilled in the art. 
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Claims 

1 . A method of measuring a performance of a route in an 
internetwork, the route coupling an internetwork server to a terminal on 

5 the internetwork, the method comprising: 

at a frequently trafficked portal on the internetwork, detecting a 
request for a web page from the terminal, wherein the web page is at 
least partially stored at the frequently trafficked portal; 

in response to the request for the web page, downloading the 
10 web page to the terminal via the internetwork; 

from the web page, retrieving a Uniform Resource Locator (URL) 
for a web object referenced in the web page; 

resolving the URL to the internetwork server; 
detecting a request for the web object from the terminal at the 
1 5 internetwork server; 

in response to the request for the web object, sending the web 
object from the internetwork server to the terminal; and 

concurrent with sending the web object, measuring a Round Trip 
Time (RTT) of one or more packets sent between the internetwork 
20 server and the terminal. 

2. The method of claim 1 , wherein the web page is at least partially 
encoded in a markup language. 

3. The method of claim 2, wherein the markup language is Hyper 
Text Markup Language. 

25 4. The method of claim 3, wherein the sending the web object from 
the internetwork server to the terminal is performed via a Hyper Text 
Transfer Protocol (HTTP), 

5. The method of claim 4, wherein the Hyper Text Transfer Protocol 
is HTTP V 1.0. 

30 

11 
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6. The method of claim 4, wherein the Hyper Text Transfer Protocol 
IS HTTP V 1.1. 

7. The method of claim 1 , wherein the web object is visually 
imperceptible. 

5 8. The method of claim 1 , wherein the web object comprises a 
single pixel. 

9. A method of measuring performance in a network, the method 
comprising: 

between a first point in the network and a second point in the 
10 network, wherein the first point is identified by a first address and the 

second point is identified by a second address, generating one or more 
pairs of packets, each of the one or more pairs of packets including: 

a packet sent from the first point to the second point; and 
a packet received at the second point from the first point, 
1 5 wherein the received packet comprises a response to the sent 
packet; 

measuring a plurality of durations between the sent packets and 
the received packets for the one or more pairs; and 

calculating, at least from the plurality of durations, parameters of 
20 at least part of the network, wherein the parameters comprise per-group 
delay, jitter, and loss. 

1 0. The method of claim 9, wherein the pairs of packets comprise 
messages in Transmission Control Protocol (TCP) format. 

1 1 . The method of claim 10, wherein one or more of the sent packets 
25 is a SYN/ACK packet. 

12. The method of claim 10, wherein one or more of the received 
packets is an ACK packet. 

13. The method of claim 9, wherein the network is an internetwork. 

12 
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14. A system for measuring performance of an internetwork, the 
system comprising: 

a frequently trafficl<ed web portal in tlie internetwork; 

a web page at least partially stored on the frequently trafficked 
5 web portal, the at least partially stored web portal including a Uniform 

Resource Locator (URL) for a web object, such that the web object is not 
stored on the frequently trafficked web portal; 

a Domain Name System (DNS) server on the internetwork; the 
DNS server including a reference which maps the URL for the web 
10 object to an Internet Protocol address for an internetwork server on the 
internetwork; 

a web browser coupled to the internetwork, wherein the web 
browser sends a download request for the web object to the internetwork 
server; and 

15 a measurement process executed on the internetwork server, 

such that in response to the download request, the measurement 
process measures one or more Round Trip Times between the 
internetwork server and the web browser. 

1 5. The system of claim 14, wherein the web page is at least partially 
20 encoded In a markup language. 

16. The system of claim 14, wherein the markup language is Hyper 
Text Markup Language (HTML). 

17. A method of measuring a performance of a route in an 
Internetwork, the route coupling an internetwork server to a terminal on 

25 the internetwork, the method comprising: 

at a frequently trafficked portal on the internetwork, detecting a 
request for a web page from the terminal, wherein the web page is at 
least partially stored at the frequently trafficked portal; 

from the web page, retrieving a Uniform Resource Locator (URL) 
30 for a web object referenced in the web page; 

resolving the URL to the internetwork server; 
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detecting a request for the web object from the terminal at the 
internetwork server; and 

in response to the request for the web object, measuring a Round 
Trip Time (RTT) of one or more packets sent between the internetwork 
5 server and the terminal. 
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